90
7
The Transmission of Information
7.7
Summary
Messages may be encoded in order to send them along a communication channel.
Shannon’s fundamental theorem proves that a message with redundancy can always
be encoded to take advantage of it, enabling a channel to transmit information up to
its maximum capacity.
The capacity of a channel is the number of symbols mm that can be transmitted in
unit time multiplied by the average information per symbol:
script upper C equals m upper I overbar periodC = m ¯I .
(7.24)
Any strategy for compressing a message is actually a search for regularities in the
message, and thus compression of transmitted information actually lies at the heart
of general scientific endeavour.
Noise added to a transmission introduces equivocation, but it is possible to transmit
information through a noisy channel with an arbitrarily small probability of error,
at the cost of lowering the channel capacity. This introduces redundancy, defined as
the quotient of the actual number of bits to the minimum number of bits necessary
to convey the information. Redundancy therefore opposes equivocation; that is, it
enables noise to be overcome. Many natural languages have considerable redundancy.
Technical redundancy arises through syntactical constraints. The degree of semantic
redundancy of English, or indeed of any other language, is currently unknown.
Problem. Attempt to define, operationally or otherwise, the terms “message”, “mes-
sage content”, and “message structure”.
Problem. Calculate the amount of information in a string of DNA coding for a
protein. Repeat for the corresponding messenger RNA and amino acid sequences. Is
the latter the same as the information contained in the final folded protein molecule?
Problem. Discuss approaches to the problem of determining the minimum quantity
of information necessary to encode the specification of an organ.
Problem. Is it useful to have a special term “bioinformation”? What would its
attributes be?
References
Ashby WR (1956) An introduction to cybernetics. Chapman and Hall, London
Benedetto D, Caglioti E, Loreto V (2002) Language trees and zipping. Phys Rev Lett 88:048702
Hamming RW (1950) Error detecting and error correcting codes. Bell Syst Tech J 26:147–160
Hartley RVL (1928) Transmission of information. Bell Syst Tech J 7:535–563
Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Statist 22:79–86
Levenshtein VI (2001) Efficient reconstruction of sequences. IEEE Trans Info Theory 47:2–22
Mandelbrot B (1952) Contribution à la théorie mathématique des jeux de communication. Publ Inst
Statist Univ Paris 2:1–124